%md-sandbox
## Instructions
In this exercise, we're doing ETL on a file we've received from some customer. That file contains data about people, including:
* first, middle and last names
* gender
* birth date
* Social Security number
* salary
But, as is unfortunately common in data we get from this customer, the file contains some duplicate records. Worse:
* In some of the records, the names are mixed case (e.g., "Carol"), while in others, they are uppercase (e.g., "CAROL").
* The Social Security numbers aren't consistent, either. Some of them are hyphenated (e.g., "992-83-4829"), while others are missing hyphens ("992834829").
The name fields are guaranteed to match, if you disregard character case, and the birth dates will also match. (The salaries will match, as well,
and the Social Security Numbers *would* match, if they were somehow put in the same format).
Your job is to remove the duplicate records. The specific requirements of your job are:
* Remove duplicates. It doesn't matter which record you keep; it only matters that you keep one of them.
* Preserve the data format of the columns. For example, if you write the first name column in all lower-case, you haven't met this requirement.
* Write the result as a Parquet file, as designated by *destFile*.
* The final Parquet "file" must contain 8 part files (8 files ending in ".parquet").
<img alt="Hint" title="Hint" style="vertical-align: text-bottom; position: relative; height:1.75em; top:0.3em" src="https://files.training.databricks.com/static/images/icon-light-bulb.svg"/> **Hint:** The initial dataset contains 103,000 records.<br/>
The de-duplicated result haves 100,000 records.
Last refresh: Never
%md
##  Classroom-Setup<br>
For each lesson to execute correctly, please make sure to run the **`Classroom-Setup`** cell at the start of each lesson (see the next cell) and the **`Classroom-Cleanup`** cell at the end of each lesson.
Classroom-Setup
For each lesson to execute correctly, please make sure to run the Classroom-Setup
cell at the start of each lesson (see the next cell) and the Classroom-Cleanup
cell at the end of each lesson.
Last refresh: Never
Last refresh: Never
Last refresh: Never